Building a Simple Agent Assistant from Scratch

This tutorial will guide you through building a knowledge base-driven intelligent Q&A Agent from scratch, suitable for enterprise internal knowledge management scenarios. This assistant can retrieve information from the enterprise knowledge base and intelligently respond to user queries, thereby improving information access efficiency.

Taking the daily work of an M365 Operations Engineer as an example, during project operations, a large number of documents often need to be consulted. These documents are diverse and scattered, making manual searching time-consuming and labor-intensive. With the help of an intelligent Agent, the efficiency of consultation can be greatly improved, reducing repetitive work and facilitating operations.

This tutorial will use this scenario as the background to demonstrate step-by-step how to build a practical knowledge Q&A assistant.

Preparing the Knowledge Base Required for the Agent

Before creating the Agent, you need to prepare and configure the knowledge base it depends on. The core capability of the Agent comes from the knowledge base content, so high-quality and well-structured knowledge documents are crucial.

In this case, we will use the existing knowledge base “Microsoft Learning Database” as the information source.

• Please ensure that documents, FAQs, operation manuals, and other content related to M365 operations have been uploaded in advance to this knowledge base.
• Supported file types generally include PDF, Word, TXT, web copies, etc. It is recommended to perform the following processing before uploading:
  • Classify and organize the content to facilitate system retrieval;
  • Use standardized file naming to avoid ambiguity;
  • Remove redundant information to improve text readability.

✅ Tip: The richness of the knowledge base content, document quality, and clarity of structure directly affect the accuracy and usability of the Agent’s answers. It is recommended that the operations team regularly update and maintain the knowledge base content.

Once the knowledge base is ready, you can connect it during the Agent configuration phase through the “Knowledge Source” feature to achieve content-based intelligent Q&A.

Creating the “M365 Support” Basic Agent

1. Go to the SERVICEME NEXT homepage, click AI Studio in the left navigation bar to enter the Agent interface.
2. In the Agent interface, click “Create Assistant” in the upper left corner, and select the “Basic Orchestration Creation” method.
3. Fill in the following basic information:
   • Assistant Name: enter M365 Support
   • Agent Avatar: select one from the system built-in avatars (custom upload is currently not supported)
   • Model Group: select the model group configured by the administrator, e.g., Standard Model Group
   • Category: select the business category, such as IT
   • Description: for example, Provide knowledge base Q&A support for M365 operations engineers
4. After filling in, click “Create” to successfully generate the basic Agent.

Configuring the “M365 Support” Basic Agent

After creation, the system will automatically enter the Agent configuration page. Complete the following configuration items in order:

1. Prompt Configuration

• Fill in brief prompt information in the Prompt input box.
• You can click “Intelligent Generate”, and the system will call the model to automatically expand the prompt, generating a more complete version.
• The prompt in this example is:

## Role
You are an M365 Operations Support Assistant, specializing in helping operations engineers analyze issues related to Microsoft 365 services and efficiently retrieving relevant resources from the knowledge base.

## Skills
1. Analyzing M365 Issues:
   - Identify and diagnose problems encountered in various M365 services such as Exchange Online, SharePoint, Teams, and OneDrive.
   - Gather relevant error messages, logs, and contextual information to accurately understand the root cause of operational issues.
   - Prioritize incidents based on severity and impact, providing clear and concise problem statements to streamline troubleshooting.

2. Retrieving Knowledge Base Resources:
   - Search and extract pertinent documentation, troubleshooting guides, and best practices from the M365 knowledge base according to the identified issue.
   - Summarize and present solutions, workarounds, and reference materials tailored to the engineer’s specific case.
   - Keep track of the most frequently used resources to optimize future query efficiency.

## Constraints
- Only address topics strictly related to M365 operations, issue analysis, and knowledge base lookup. Decline unrelated queries.
- All outputs must follow the prescribed structure and remain within the context of technical support for Microsoft 365.
- Do not provide generic advice or speculative troubleshooting steps without referencing official or knowledge base resources.

2. Greeting Configuration

• You can fill in a custom greeting or click “Intelligent Generate” to automatically generate a welcome message.
• The greeting in this example is:

Hello, I am M365 Support, here to assist you with Microsoft 365-related questions and support needs.

[How can you help me troubleshoot issues with Microsoft 365?] [What are some common problems users face with M365 and how can I resolve them?] [Can you guide me on optimizing my use of Microsoft 365 tools and features?]

3. Model Group Settings

• The model group selected during Agent creation (e.g., Standard Model Group) will be automatically populated here.

• You can switch as needed, noting:

  • Model group contents may differ across environments;
  • Model groups are pre-configured by administrators;
  • The Standard Model Group used in this example includes models: gpt-4.1, DeepseekR1-Ai/DeepSeek, gpt-4.1-mini.

  

4. Skill Configuration (Optional)

• Skills must be created in advance by administrators in skill management before they can be used.
• Common skills include: web reading, news retrieval, etc.
• In this example, no skills are configured. It is recommended to add skills only when necessary, as too many skills may affect Agent performance.

5. Conversation Experience Settings

Conversation experience determines the detailed interaction behavior between the Agent and users. It is recommended to configure reasonably according to the business scenario.

• Number of Context Memory Entries:
  Set to 5, meaning the Agent will retain the last 5 rounds of conversation content for contextual understanding.
  • Advantages: improves multi-turn dialogue coherence, facilitates handling complex queries.
  • Note: too much context may slow response or cause information confusion; 5 is a generally recommended value.

• Click the gear icon to enter advanced settings. Common options include:

  • Enable User Suggested Questions

    • Function: automatically recommend follow-up questions to users based on the current conversation context.
    • Suitable scenarios: enhances user query guidance, friendly for novice users.
    • Recommendation for this case: can be disabled, as operations engineers usually have clear goals.

  • Enable Question Guidance ✅

    • Function: displays example questions in the input box to reduce user query difficulty.
    • Recommendation for this case: enable, helps users understand the scope of possible questions.

  • Enable Chat History ✅

    • Function: retains historical dialogue records between users and the Agent for later review.
    • Suitable scenarios: issue tracking, case reproduction, knowledge base tuning.
    • Recommendation for this case: enable, convenient for operations personnel to trace issues.

  • Enable Chat Feedback ✅

    • Function: allows users to like/dislike or leave comments on each answer to collect feedback.
    • Suitable scenarios: monitoring and continuous optimization of answer quality.
    • Recommendation for this case: enable, facilitates collecting user opinions and improving Agent answer quality.

  • Enable Keyword Filtering

    • Function: intercepts inputs/outputs containing sensitive words to ensure content compliance.
    • Recommendation for this case: enable as appropriate according to internal enterprise compliance requirements.

6. Knowledge Source Configuration

The knowledge source determines the core information source for the Agent’s answers. Correct configuration of knowledge base and retrieval strategy can significantly improve Q&A accuracy.

• Select Knowledge Base
  • Click the “+” on the right, and select the required knowledge base from the pop-up list, such as:
    Microsoft Learning Database (a collection of M365 operations-related documents).

• Configure Knowledge Source (gear icon)
  Enter the knowledge base settings interface for detailed parameter configuration:

  • Retrieval Strategy: Hybrid Retrieval

    • Hybrid retrieval combines semantic understanding and keyword matching, improving recall rate and accuracy.
    • Recommended to enable in this case, suitable for knowledge Q&A scenarios with large document volumes and diverse query forms.

  • Maximum Recall Number: 5

    • The system recalls up to 5 most relevant contents from the knowledge base each time for answer generation.
    • Recommended value is 5; too high slows response, too low may miss key information.

  • Force Private Domain Q&A: Enabled ✅

    • When enabled, the Agent will only use the selected knowledge base content to generate answers, no longer referencing general model knowledge.
    • Advantage: ensures answers are factual, accurate, and controllable.
    • Recommendation for this case: enable to avoid generating generalized content unrelated to M365.

  • Document Matching Similarity, QnA Matching Similarity: Keep default

    • Default settings suit most scenarios. If recall failures or misjudgments occur during actual use, adjust this value appropriately (usually between 0.6 ~ 0.8).

⚠️ This scenario does not require configuring external data sources. If there is no need to connect real-time interfaces or databases, you can skip the “Data Source” configuration.

7. Save and Test

• After configuration, a prompt “Assistant edited, please click sync” will appear at the top right.
• Click “Sync”, then you can test Q&A on the right side by entering queries such as:
  • “How to create a Copilot?”
  • “How to configure Copilot?”
• Adjust the configuration based on test results until the answers meet expectations.
• Finally, click “Publish” at the top right to complete the Agent configuration.

Complete configuration as follows: